refactor: read and write config toml in TypeScript#3662
refactor: read and write config toml in TypeScript#3662christierney wants to merge 27 commits intomainfrom
Conversation
| * | ||
| * Skips: | ||
| * - Dot-directories (except .posit itself) | ||
| * - node_modules, __pycache__, renv, packrat |
There was a problem hiding this comment.
The Go code has a more complicated setup for excluding files: see https://github.com/posit-dev/publisher/blob/main/internal/bundles/matcher/walker.go
That code is also used for excluding files from bundles, etc. It didn't seem worth implementing all that for just loading configs and the above skiplist seemed sufficient to me, but I'm interested in feedback on this point.
There was a problem hiding this comment.
the argument makes sense to me - this is specific to config loading
| * Filters redundant unevaluatedProperties errors when a more specific | ||
| * error exists at the same or deeper path (matching Go's behavior). | ||
| */ | ||
| export function formatValidationErrors(errors: ErrorObject[]): string { |
There was a problem hiding this comment.
This function is attempting to produce useful error messages from TOML validation failures, and to make them match what the Go validator returns (e.g. <key>: not allowed or <key>: missing property). The e2e tests look for these specific error messages. We could decide to let the messages from ajv through if we want, but this keeps the behavior and test changes smaller. In any case we need to do some processing to extract actionable info from the array of validation errors, even if we don't coerce the messages into the Go form.
| const rawConfigs = response.data; | ||
| // remove the errors | ||
| configurations = configurations.filter( | ||
| (cfg): cfg is Configuration => !isConfigurationError(cfg), |
There was a problem hiding this comment.
this was redundant as filterConfigurationToValidAndType already filters out errors
56943cb to
745207f
Compare
| selectedInspectionResult.configuration.productType = getProductType( | ||
| activeDeployment.serverType, | ||
| ); |
There was a problem hiding this comment.
this is a fix for a pre-existing bug I found while testing. This path was not setting a product type, so you'd end up with an invalid config TOML (with 'product_type = '').
The new deployment flow already does this:
publisher/extensions/vscode/src/multiStepInputs/newDeployment.ts
Lines 856 to 857 in 9367e0f
| on: | ||
| pull_request: | ||
| workflow_dispatch: | ||
| inputs: |
There was a problem hiding this comment.
thanks for fixing this!
|
|
||
| /** | ||
| * Compute a relative projectDir from an absolute path, using "." for the root. | ||
| * Matches Go's convention where projectDir is relative to the workspace root. |
There was a problem hiding this comment.
Go reference out of date?
There was a problem hiding this comment.
I don't think it's out of date, but maybe not very useful? The point was that we have to match what the Go API expects (use a relative path) when calling it with a directory argument. So when we construct a Configuration object we also give it a relative projectDir that ends up getting passed to a bunch of API calls.
There was a problem hiding this comment.
ah, I see - yeah, I wonder if we'll end up with a bunch of out of date Go comments by the end of this that we'll have to sweep out (which is fine)
There was a problem hiding this comment.
Yeah my feeling is that they will provide guidance for us and Claude while the migration is ongoing, and then we can sweep through and delete them.
There was a problem hiding this comment.
If we need to remember to do that, can we have a "clean up" issue where we make a note of this? Don't want to forget, there's a lot changing.
There was a problem hiding this comment.
I'll start adding specific line references to this
| return rel === "" ? "." : rel; | ||
| } | ||
|
|
||
| async function loadConfigsFromPaths( |
There was a problem hiding this comment.
maybe useful to have code comments for this helper? The Promise of an array of Config | ConfigurationError is confusing if you didn't read about it in a different code comment.
| const relDir = relativeProjectDir(dir, rootDir); | ||
| const configPaths = await listConfigFiles(dir); | ||
| const configs = await loadConfigsFromPaths(configPaths, relDir); | ||
| results.push(...configs); |
There was a problem hiding this comment.
this would be silly, but what would be the correct behavior if you have nested .posit directories?
There was a problem hiding this comment.
IMO if you have a .posit in your .posit you are doing something nonstandard and we should not support it
There was a problem hiding this comment.
Agreed I wouldn't expect to see a .posit dir inside a .posit dir at any point.
Add a TypeScript-native TOML configuration loader that matches Go's config.FromFile behavior, as a building block for migrating away from the Go backend. - Add smol-toml, ajv, ajv-formats dependencies for TOML parsing and JSON Schema validation - Create src/toml/ module: loader, key converter, error factories, and JSON schema copy - Consolidate TS types to match Go and schema (remove dead types/fields, add missing ones) - Remove dead Go types (Schedule, AccessType, ConnectAccessControl) and unreachable code in content.go - Fix schema: auth_type in integration_requests (was camelCase), add additionalProperties: false to integration_requests items Fixes #3651 Fixes #3652 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Jordan Jensen <jordan.jensen@posit.co>
Complete the toml module with: - convertKeysToSnakeCase: reverse of camelCase converter for writing TOML - forceProductTypeCompliance: port of Go's compliance logic for Connect Cloud - writeConfigToFile: validates, transforms, and writes config TOML files - discovery functions: listConfigFiles, loadConfiguration, loadAllConfigurations, loadAllConfigurationsRecursive for finding and loading configs from disk - barrel exports in index.ts Also removes unnecessary type assertions across test files, replacing `as ConfigurationLoadError` with `instanceof` checks per project conventions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all callsites of api.configurations.get, getAll, and createOrUpdate with direct calls to the TypeScript toml module: - state.ts: getSelectedConfiguration uses loadConfiguration, refreshConfigurations uses loadAllConfigurationsRecursive - newDeployment.ts: uses loadAllConfigurations + writeConfigToFile - selectNewOrExistingConfig.ts: uses loadAllConfigurations + writeConfigToFile - homeView.ts: uses loadAllConfigurations with entrypoint filtering Remove get, getAll, and createOrUpdate from Configurations API class (inspect is kept as it still requires the Go backend). Update state.test.ts to mock toml module instead of API client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The toml module was storing absolute paths in Configuration.projectDir, but the Go backend's ProjectDirFromRequest resolves dir query parameters relative to the workspace root. This caused 404 errors from Go API endpoints (files, secrets, packages, etc.) that still receive projectDir. Discovery functions now accept a rootDir parameter and compute relative paths (e.g., "." or "subdir") for Configuration metadata, while using absolute paths internally for file I/O. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… + rootDir writeConfigToFile now takes (configName, projectDir, rootDir, config) instead of (configPath, projectDir, config), matching the signature pattern of loadConfiguration and loadAllConfigurations. This eliminates the need for callers to compute absolute paths and call getConfigPath separately, removing a class of path-semantic mismatches. getConfigPath is no longer exported. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove internal-only symbols (loadConfigFromFile, forceProductTypeCompliance, convertKeysToCamelCase, convertKeysToSnakeCase, getConfigDir, listConfigFiles) from the barrel file. These remain exported on their source files for intra-module use and direct test imports, but are no longer part of the module's public API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
stripEmpty was removing parent objects (like `r: {}`) after stripping
their empty-string children. The JSON schema conditionally requires
these sections (e.g., `r` for R content types), so removing them caused
validation failures when creating new R Shiny configurations.
Go's TOML encoder writes section headers like `[r]` even when all
fields are omitted via omitempty. Match that behavior by only stripping
leaf values (undefined, null, empty strings), never parent objects.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tions Error location strings in getSummaryStringFromError still referenced the removed Go API methods (configurations.getAll, configurations.createOrUpdate). Updated to reference the toml module functions that replaced them. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These endpoints are no longer called by the extension — configuration
reading and writing is now handled by the TypeScript toml module.
Deletes:
- GET /api/configurations (GetConfigurationsHandlerFunc)
- GET /api/configurations/{name} (GetConfigurationHandlerFunc)
- PUT /api/configurations/{name} (PutConfigurationHandlerFunc)
Retains configDTO and configLocation types in get_configurations.go
as they are still used by other handlers (files, secrets, integration
requests).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These directories can be large in Python/R projects and will never contain .posit/publish/ configuration files. Skipping them avoids unnecessary filesystem traversal. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Format ajv validation errors with "key: problem" style matching Go's jsonschema library (e.g., "invalidParam: not allowed." instead of "must NOT have unevaluated properties"). Also drop the file path prefix from the error msg field, matching Go's AgentError format. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ionErrors Port Go's entrypointObjectRef handling to TypeScript compliance: - For Connect, copy entrypointObjectRef to entrypoint (object-reference style) - Always clear entrypointObjectRef before validation (non-TOML field) - Delete entrypointObjectRef in writer alongside comments/alternatives Move formatValidationErrors from loader.ts to errors.ts so both loader and writer produce Go-compatible schema error messages. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The redundancy filter was using pathKey (the parent object path from ajv's instancePath) for prefix comparison. For root-level errors, pathKey is "" which matches everything, incorrectly dropping unevaluatedProperties errors whenever any other error existed. Use fullKey (including the property name) to match Go's behavior, where InstanceLocation points to the property itself. Now only a deeper error at "python.garbage.something" filters out the "python.garbage: not allowed." error. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…checks workspaces.path() returns string | undefined. Four call sites used the non-null assertion (!) assuming a workspace must be open. Replace with explicit undefined checks and early returns, matching the existing pattern in state.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… depth limit Parallelize config file loading with Promise.allSettled, eliminate an extra fs.stat per directory by checking readdir entries, remove the abs→rel→abs path round-trip in walkForConfigs, and add a depth limit (20) to prevent runaway walks in pathological directory trees. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… last Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a local loadError closure that captures location, replacing 4 repeated new ConfigurationLoadError(createConfigurationError(..., location)) patterns with single-line throw statements. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extract shared convertKeys() helper that both convertKeysToCamelCase and convertKeysToSnakeCase delegate to, eliminating near-identical code. In writer.ts, reuse getConfigPath from discovery and add loadError closure matching the loader.ts pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The "Create New Configuration For Destination" command was writing configs with product_type = '' because the Go inspect API doesn't populate productType. The newDeployment flow already derived it from the credential's serverType via getProductType — apply the same pattern here using activeDeployment.serverType. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
4bd46f9 to
ae8edc6
Compare
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| const existingNames = ( | ||
| await api.configurations.getAll(selectedInspectionResult.projectDir) | ||
| ).data.map((config) => config.configurationName); | ||
| const root = workspaces.path(); |
There was a problem hiding this comment.
would you mind explaining what this workspaces.path() call is doing? Are we just checking that a workspace path exists at all?
There was a problem hiding this comment.
So, the Go API is given the workspace root at startup:
publisher/extensions/vscode/src/servers.ts
Lines 46 to 53 in 4f49e98
And then all of the file handling is done relative to that root. Various API calls expect a "projectDir" path argument that is relative to that root. This gets stored on the Configuration object so that the extension can pass it to the Go APIs.
So any code that loads a config needs to also tell the loader what workspace root it is working from, so the loader can correctly determine relProjectPath.
The other alternative I considered was making a ConfigurationLoader object and injecting it into every part of the extension that needed it, and that singleton would have had an internal reference to the workspace root. But that ended up looking messier.
There was a problem hiding this comment.
I'm glad this got asked, because I had the same exact question as you can see by my other comment.
There was a problem hiding this comment.
If there's a less confusing way to implement this I'm definitely open to it! Somebody needs to look up the workspace root but it doesn't necessarily have to be done this way.
| // Keys at these paths contain user-defined names that must not be converted. | ||
| const PRESERVE_KEYS_PATHS = new Set(["environment"]); | ||
|
|
||
| // Both camelCase and snake_case forms are checked because the parentKey |
There was a problem hiding this comment.
can you say a little more about what's going on with this case? The code comment is confusing me a bit.
There was a problem hiding this comment.
I originally had completely separate functions for convertKeysToCamelCase and convertKeysToSnakeCase, but they were almost identical, so I extracted the common code into convertKeys. That function needs to check "is the current field I'm looking at the Integration Requests field?" and it needs to work no matter which form (snake_case or camelCase) the object is currently in, so we check both versions of the field name.
There was a problem hiding this comment.
I agree I had to read the comment twice, then come to this explanation.
There was a problem hiding this comment.
I'll try to rework this comment.
| } | ||
|
|
||
| // Content types that have a mapping in Connect Cloud. | ||
| // Keep in sync with Go's CloudContentTypeFromPublisherType in internal/clients/types/types.go |
There was a problem hiding this comment.
here's stuff where we probs want some type of tracking ticket to remember to delete these comments - they're useful now but they won't always be.
| } | ||
| const allConfigs = await loadAllConfigurations( | ||
| selectedInspectionResult.projectDir, | ||
| root, |
There was a problem hiding this comment.
@zackverham here's where we use the root
dotNomad
left a comment
There was a problem hiding this comment.
This is looking great! I had a few questions and nits. The only thing that needs to be changed prior to merge is the CHANGELOG addition needs to go into Unreleased
| ### Fixed | ||
|
|
||
| - Connect Cloud users who have permissions to publish to multiple accounts are able to create credentials again. (#3446) | ||
| - Fixed configuration schema to use `auth_type` (snake_case) for integration requests, matching the format the extension actually produces. Added strict property validation to integration request items. (#3651) |
There was a problem hiding this comment.
This needs to go into the Unreleased section.
There was a problem hiding this comment.
gah! I think this may have happened upon rebasing.
| // Keys at these paths contain user-defined names that must not be converted. | ||
| const PRESERVE_KEYS_PATHS = new Set(["environment"]); | ||
|
|
||
| // Both camelCase and snake_case forms are checked because the parentKey |
There was a problem hiding this comment.
I agree I had to read the comment twice, then come to this explanation.
| if (!root) { | ||
| return getDeploymentObjects(); | ||
| } |
There was a problem hiding this comment.
Can you explain this new code to me?
It looks like it gets the path of the workspace folder, but if there isn't a workspace folder it gets the deployment objects?
I guess this happened prior if api.configurations.getAll failedbecause the catch returns getDeploymentObjects(), but loadAllConfigurations no longer fails if we don't have a workspace folder open?
There was a problem hiding this comment.
the getDeploymentObjects() thing threw me too, but this seems to be what the functions in this file do upon error. See all the search results for that func: https://github.com/posit-dev/publisher/blob/main/extensions/vscode/src/multiStepInputs/newDeployment.ts#L129
| const existingNames = ( | ||
| await api.configurations.getAll(selectedInspectionResult.projectDir) | ||
| ).data.map((config) => config.configurationName); | ||
| const root = workspaces.path(); |
There was a problem hiding this comment.
I'm glad this got asked, because I had the same exact question as you can see by my other comment.
| const relDir = relativeProjectDir(dir, rootDir); | ||
| const configPaths = await listConfigFiles(dir); | ||
| const configs = await loadConfigsFromPaths(configPaths, relDir); | ||
| results.push(...configs); |
There was a problem hiding this comment.
Agreed I wouldn't expect to see a .posit dir inside a .posit dir at any point.
| rootDir: string, | ||
| depth: number = 20, | ||
| ): Promise<(Configuration | ConfigurationError)[]> { | ||
| if (depth <= 0) return []; |
There was a problem hiding this comment.
I expected this to look at this directory for configs, not no looking at all. I could get used to it though, so small nit.
There was a problem hiding this comment.
Not sure I follow you. Are you saying the way the recursion terminates is confusing/surprising?
| name === "node_modules" || | ||
| name === "__pycache__" || | ||
| name === "renv" || | ||
| name === "packrat" |
There was a problem hiding this comment.
Could be handy to turn this into a const array and check if name is in that array.
Intent
Migrate configuration reading and writing from Go API calls to a pure TypeScript toml module in the VSCode extension. This removes three HTTP round-trips to the Go backend for operations that only need local file I/O, and moves us toward reducing the extension's dependency on the Go subprocess.
Fixes #3651
Fixes #3652
Fixes #3653
Type of Change
Approach
Updated type definitions and the schema to match current behavior:
Added a new
src/toml/module with seven files:ConfigurationLoadErrorclass and error factory functionsenvironmentandintegration_requests[].configForceProductTypeCompliance()for Connect CloudlistConfigFiles), single/batch/recursive loading, path resolutionThe public API surface is five exports:
loadConfiguration,loadAllConfigurations,loadAllConfigurationsRecursive,writeConfigToFile, andConfigurationLoadError. Internal helpers stay exported on their source files for intra-module use and tests, but are not re-exported from the barrel.All functions that produce
Configurationobjects accept(projectDir, rootDir)whereprojectDiris relative (e.g.,"."or"subdir") androotDiris the absolute workspace root. Path resolution to absolute paths for file I/O happens inside the module. This matches the Go convention whereProjectDirFromRequestresolves relativedirparameters against the workspace base.After replacing all callsites, the three Go API handlers (
GET /configurations,GET /configurations/{name},PUT /configurations/{name}) and their tests were deleted. The sharedconfigDTO/configLocationtypes remain as they're used by other endpoints.A copy of the JSON schema is bundled at
src/toml/schemas/posit-publishing-schema-v3.json, identical to the Go source of truth. A comment in the schema test notes the need to keep them in sync.Key design decisions:
Configuration— no write-then-read round-tripsrc/toml/has no vscode imports — it's pure Node.jsstripEmptyonly removes leaf values (undefined, null, empty strings), never parent objects, because the schema conditionally requires section headers like[r]even when all fields are emptyUser Impact
No user-facing behavior change. Configuration reading and writing now happens directly in the extension process instead of through HTTP calls to the Go backend.
Automated Tests
[ed: there's probably more now, I've made some improvements since ]
a bunch of new tests in
src/toml/:Existing
state.test.tstests were rewritten to mock the toml module instead of the API client.Directions for Reviewers
I recommend reading through the new
toml/directory first and then reviewing the updates to the extension code.I am working my way through these manual tests:
[python]section, snake_case keys[r]section, no schema errorChecklist
yes, only for the auth_type schema fix. The rest should not be user-facing.